1.1.1 Logic

We have group \(A\) that enters treatment at some point and group \(B\) that never does

The estimate:

\[\hat\tau = (\mathbb{E}[Y^A | post] - \mathbb{E}[Y^A | pre]) -(\mathbb{E}[Y^B | post] - \mathbb{E}[Y^B | pre])\] (how different is the change in \(A\) compared to the change in \(B\)?)

can be written:

\[\hat\tau = (\mathbb{E}[Y_1^A | post] - \mathbb{E}[Y_0^A | pre]) -(\mathbb{E}[Y_0^B | post] - \mathbb{E}[Y_0^B | pre])\]

1.1.2 Logic

Cleaning up

\[\hat\tau = (\mathbb{E}[Y_1^A | post] - \mathbb{E}[Y_0^A | pre]) -(\mathbb{E}[Y_0^B | post] - \mathbb{E}[Y_0^B | pre])\]

\[\hat\tau = (\mathbb{E}[Y_1^A | post] - \mathbb{E}[Y_0^A | post]) + ((\mathbb{E}[Y_0^A | post] - \mathbb{E}[Y_0^A | pre]) -(\mathbb{E}[Y_0^B | post] - \mathbb{E}[Y_0^B | pre]))\]

\[\hat\tau_{ATT} = \tau_{ATT} + \text{Difference in trends}\]

1.1.4 Diagnosis

Here only the two way fixed effects is unbiased and only for the ATT.

The ATT here is averaging over effects for treated units (later periods only). We know nothing about the size of effects in earlier periods when all units are in control!

design |> diagnose_design() 

1.1.6 Extends to multiple units easily

design |> redesign(n_units = 10) |> diagnose_design() 

1.1.9 Heterogeneity

• Things get much more complicated when there is (a) heterogeneous timing in treatment take up and (b) heterogeneous effects

• It’s only recently been appreciated how tricky things can get

• But we already have an intuition from our analysis of trials with heterogeneous assignment and heterogeneous effects:

• in such cases fixed effects analysis weights stratum level treatment effects by the variance in assignment to treatment

• something similar here

1.1.10 Staggared assignments

Just two units assigned at different times:

trend = 0

design <- 
  declare_model(
    unit = add_level(N = 2, ui = rnorm(N), I = 1:N),
    time = add_level(N = 6, ut = rnorm(N), T = 1:N, nest = FALSE),
    obs = cross_levels(by = join_using(unit, time))) +
  declare_model(
    potential_outcomes(Y ~ trend*T + (1+Z)*(I == 2))) +
  declare_assignment(Z = 1*((I == 1) * (T>3) + (I == 2) * (T>5))) +
  declare_measurement(Y = reveal_outcomes(Y~Z), 
                      I_c = I - mean(I)) +
  declare_inquiry(mean(Y_Z_1 - Y_Z_0)) +
  declare_estimator(Y ~ Z, label = "1. naive") + 
  declare_estimator(Y ~ Z + I, label = "2. FE1") + 
  declare_estimator(Y ~ Z + as.factor(T), label = "3. FE2") + 
  declare_estimator(Y ~ Z + I + as.factor(T), label = "4. FE3") + 
  declare_estimator(Y ~ Z*I_c + as.factor(T), label = "5. Sat")  

1.1.11 Staggared assignments diagnosis

1.1.14 Where do these numbers come?

• The estimate when we control for time is -1: this comes from weighting the time-stratum level effects according to the variance of assignment to each stratum
• it weights periods 4 and 5 only and equally, yielding the difference in outcomes for unit 1 in treatment (0) and group 2 in control (1)

design |> draw_data()  |> group_by(time) |> summarize(var = mean(Z)*(1-mean(Z))) |>
  mutate(weight = var/sum(var)) |> kable()

1.1.25 Triple differencing

You are interested in the effects of influx of refugees on right wing voting

• You have (say more conservative) states with no refugees at any period
• You have (say more liberal) states with refugees post 2016 only

You want to do differences in differences comparing these states before and after

• However you worry that things change differntially in the conservative and liberal states: no parallel trends

• but you can identify areas within states that are more or less likely to be exposed and compare differnces in differences in the exposed and unexposed groups.

1.1.26 Triple differencing

So:

• Two types of states: \(L \in \{0,1\}\), only \(L=1\) types get refugee influx
• Two time periods: \(Post \in \{0,1\}\), refugee influx occurs in period \(Post = 1\)
• Two groups: \(B \in \{0,1\}\), \(B=1\) types affected by refugee influx

\[Y = \beta_0 + \beta_1 L + \beta_2 B + \beta_3 Post + \beta_4 LB + \beta_5 L Post + \beta_6 B Post + \beta_7L B Post + \epsilon\]

1.2.4 The list experiment: Strategy

• Respondents are given a short list and a long list.

• The long list differs from the short list in having one extra item—the sensitive item

• We ask how many items in each list does a respondent agree with:

  • \(Y_i(0)\) is the number of elements on a short list that a respondent agrees with
  • \(Y_i(1)\) is the number of elements on a long list that a respondent agrees with
  • \(Y_i(1) - Y_i(0)\) is an indicator for whether an individual agrees with the sensitive item
  • \(\mathbb{E}[Y_i(1) - Y_i(0)]\) is the share of people agreeing with sensitive item

1.2.5 The list experiment: Simplified example

How many of these do you agree with:

[Note: this is obviously not a good list. Why not?]

1.2.14 Individual or group effects?

• This is typically used to estimate average levels

• However you can use it in the obvious way to get average levels for groups: this is equivalent to calculating group level heterogeneous effects

• Extending the idea you can even get individual level estimates: for instance you might use causal forests

• You can also use this to estimate the effect of an experimental treatment on an item that’s measured using a list, without requiring individual level estimates:

\[Y_i = \beta_0 + \beta_1Z_i + \beta_2Long_i + \beta_3Z_iLong_i\]

1.3.5 Bandwidth tradeoff

rdd_design |> 
  redesign(bandwidth = seq(from = 0.05, to = 0.5, by = 0.05)) |> 
  diagnose_designs()

• As we increase the bandwidth, the lm bias gets worse, but slowly, while the error falls.
• The best bandwidth is relatively wide.
• This is more true for the optimal estimator.

1.4.1 LATE—Local Average Treatment Effects

Sometimes you give a medicine but only a nonrandom sample of people actually try to use it. Can you still estimate the medicine’s effect?

Say that people are one of 3 types:

1. \(n_a\) “always takers” have \(X=1\) no matter what and have average outcome \(\overline{y}_a\)
2. \(n_n\) “never takers” have \(X=0\) no matter what with outcome \(\overline{y}_n\)
3. \(n_c\) “compliers have” \(X=Z\) and average outcomes \(\overline{y}^1_c\) if treated and \(\overline{y}^0_c\) if not.

1.4.2 LATE—Local Average Treatment Effects

Sometimes you give a medicine but only a non random sample of people actually try to use it. Can you still estimate the medicine’s effect?

We can figure something about types:

1.4.3 LATE—Local Average Treatment Effects

You give a medicine to 50% but only a non random sample of people actually try to use it. Can you still estimate the medicine’s effect?

Average in \(Z=0\) group: \(\frac{{n_c} \overline{y}^0_{c}+ \left(n_{n}\overline{y}_{n} +{n_a} \overline{y}_a\right)}{n_a+n_c+n_n}\)

Average in \(Z=1\) group: \(\frac{{n_c} \overline{y}^1_{c} + \left(n_{n}\overline{y}_{n} +{n_a} \overline{y}_a \right)}{n_a+n_c+n_n}\)

Difference: \(ITT = ({\overline{y}^1_c-\overline{y}^0_c})\frac{n_c}{n} \rightarrow LATE = ITT\times\frac{n}{n_c}\)

1.4.4 The good and the bad of LATE

• You get a well-defined estimate even when there is non-random take-up
• May sometimes be used to assess mediation or knock-on effects
• But:
  • You need assumptions (monotonicity and the exclusion restriction – where were these used above?)
  • Your estimate is only for a subpopulation
  • The subpopulation is not chosen by you and is unknown
  • Different encouragements may yield different estimates since they may encourage different subgroups

1.5.2 SUTVA violations

More completely specified potential outcomes (and estimands)

Table: Potential outcomes for four units for different treatment profiles, \(D_1\)-\(D_4\). \(D_i\) represents an allocation to treatment and \(y_j(D_i)\) is the potential outcome for (row) unit \(j\) given (column) allocation \(i\).

1.5.3 SUTVA violations

• The key is to think through the structure of spillovers.
• Here immediate neighbors are exposed
• In this case we can define a direct treatment (being exposed) and an indirect treatment (having a neighbor exposed) and we can work out the propensity for each unit of receiving each type of treatment
• These may be non uniform (here central types are more likely to have teated neighbors); but we can still use the randomization to assess effects

Idea: You can use the design to get a handle on spillovers

1.6.2 The problem of unidentified mediators

1.6.3 The problem of unidentified mediators

An obvious approach is to first examine the (average) effect of X on M1 and then use another manipulation to examine the (average) effect of M1 on Y.

• But both of these average effects may be positive (for example) even if there is no effect of X on Y through M1.

1.6.4 The problem of unidentified mediators

An obvious approach is to first examine the (average) effect of X on M1 and then use another manipulation to examine the (average) effect of M1 on Y.

• Similarly both of these average effects may be zero even if X affects on Y through M1 for every unit!

1.6.5 The problem of unidentified mediators

• Another somewhat obvious approach is to see how the effect of \(X\) on \(Y\) in a regression is reduced when you control for \(M\).

• If the effect of \(X\) on \(Y\) passes through \(M\) then surely there should be no effect of \(X\) on \(Y\) after you control for \(M\).

• This common strategy associated with Baron and Kenny (1986) is also not guaranteed to produce reliable results. See for instance Green, Ha, and Bullock (2010)

1.6.6 Baron Kenny issues

df <- 
  fabricate(N = 1000,
            U = rbinom(N, 1, .5),
            X = rbinom(N, 1, .5),
            M = ifelse(U==1, X, 1-X),
            Y = ifelse(U==1, M, 1-M)) 
            
list(lm(Y ~ X, data = df), 
     lm(Y ~ X + M, data = df)) |> texreg::htmlreg()

1.6.7 The problem of unidentified mediators

• See Imai on better ways to think about this problem and designs to address it.

1.6.8 The problem of unidentified mediators: Quantities

• In the potential outcomes framework we can describe a mediation effect as (see Imai et al): \[\delta_i(t) = Y_i(t, M_i(1)) - Y_i(t, M_i(0)) \textbf{ for } t = 0,1\]
• The direct effect is: \[\psi_i(t) = Y_i(1, M_i(t)) - Y_i(0, M_i(t)) \textbf{ for } t = 0,1\]
• This is a decomposition, since: \[Y_i(1, M_i(1)) - Y_1(0, M_i(0)) = \frac{1}{2}(\delta_i(1) + \delta_i(0) + \psi_i(1) + \psi_i(0)) \]
• If (and a big if), there are no interaction effects—ie \(\delta_i(1) = \delta_i(0), \psi_i(1) = \psi_i(0)\), then \[Y_i(1, M_i(1)) - Y_1(0, M_i(0)) = \delta_i + \psi_i\]
• The bad news is that although a single experiment might identify the total effect, it can not identify these elements of the direct effect.

1.6.9 The problem of unidentified mediators: Solutions?

• Check formal requirement for identification under single experiment design (“sequential ignorability”—that, conditional on actual treatment, it is as if the value of the mediation variable is randomly assigned relative to potential outcomes). But this is strong (and in fact unverifiable) and if it does not hold, bounds on effects always include zero (Imai et al)
• You can use interactions with covariates if you are willing to make assumptions on no heterogeneity of direct treatment effects over covariates. eg you think that money makes people get to work faster because they can buy better cars; you look at the marginal effect of more money on time to work for people with and without cars and find it higher for the latter. This might imply mediation through transport but only if there is no direct effect heterogeneity (eg people with cars are less motivated by money).